Generating fused sensor data through metadata association

ABSTRACT

Described herein are systems, methods, and non-transitory computer readable media for generating fused sensor data through metadata association. First sensor data captured by a first vehicle sensor and second sensor data captured by a second vehicle sensor are associated with first metadata and second metadata, respectively, to obtain labeled first sensor data and labeled second sensor data. A frame synchronization is performed between the first sensor data and the second sensor data to obtain a set of synchronized frames, where each synchronized frame includes a portion of the first sensor data and a corresponding portion of the second sensor data. For each frame in the set of synchronized frames, a metadata association algorithm is executed on the labeled first sensor data and the labeled second sensor data to generate fused sensor data that identifies associations between the first metadata and the second metadata.

The present invention relates generally to generating fused datasets,and more particularly, in some embodiments, to generating fused sensordata by associating metadata corresponding to first sensor data withmetadata corresponding to second sensor data.

BACKGROUND

On-board sensors in a vehicle, such as an autonomous vehicle (AV),supplement and bolster the vehicle's FOV by providing continuous streamsof sensor data captured from the vehicle's surrounding environment.Sensor data is used in connection with a diverse range of vehicle-basedapplications including, for example, blind spot detection, lane changeassisting, rear-end radar for collision warning or collision avoidance,park assisting, cross-traffic monitoring, brake assisting, emergencybraking, and automated distance control.

On-board vehicle sensors may include, for example, cameras, lightdetection and ranging (LiDAR)-based systems, radar-based systems, GlobalPositioning System (GPS) systems, sonar-based sensors, ultrasonicsensors, inertial measurement units (IMUs), accelerometers, gyroscopes,magnetometers, and far infrared (FIR) sensors. Sensor data may includeimage data, reflected laser data, or the like. Often, images captured byon-board sensors utilize a three-dimensional (3D) coordinate system todetermine the distance and angle of objects in the image with respect toeach other and with respect to the vehicle. In particular, suchreal-time spatial information may be acquired near a vehicle usingvarious on-board sensors located throughout the vehicle, which may thenbe processed to calculate various vehicle parameters and determine safedriving operations of the vehicle.

Sensor data captured by different on-board sensors may be associatedwith various types of metadata. Such metadata may be associated atdifferent points in time with sensor data captured by different sensors.This may make it difficult to establish a correspondence between sensordata from different sensors despite such sensor data relating to a samecaptured scene. Discussed herein are technical solutions that addressthis and other technical drawbacks associated with conventionaltechniques for manipulating sensor data.

SUMMARY

In an example embodiment, a computer-implemented method for fusingsensor data via metadata association is disclosed. Thecomputer-implemented method includes capturing first sensor data using afirst vehicle sensor and second sensor data using a second vehiclesensor and associating first metadata with the first sensor data toobtain labeled first sensor data and second metadata with the secondsensor data to obtain labeled second sensor data. The method furtherincludes performing a frame synchronization between the labeled firstsensor data and the labeled second sensor data to obtain a set offrames, where each frame includes a respective portion of the labeledfirst sensor data and a corresponding respective portion of the labeledsecond sensor data. The method additionally includes executing, for eachframe in the set of frames, a metadata association algorithm to thelabeled first sensor data and the labeled second sensor data andgenerating, based at least in part on an output of executing themetadata association algorithm, fused sensor data from the labeled firstsensor data and the labeled second sensor data.

In an example embodiment, the first sensor data is three-dimensional(3D) point cloud data and the second sensor data is two-dimensional (2D)image data.

In an example embodiment, associating the first metadata with the firstsensor data to obtain the labeled first sensor data includes associatinga 3D bounding box with a first object present in the 3D point clouddata, and associating the second metadata with the second sensor data toobtain the labeled second sensor data includes associating a 2D boundingbox with a second object present in the 2D image data.

In an example embodiment, associating the first metadata with the firstsensor data to obtain the labeled first sensor data and the secondmetadata with the second sensor data to obtain the labeled second sensordata further includes associating a first classification with the firstobject and a second classification with the second object.

In an example embodiment, each synchronized frame is associated with i)a respective set of 3D labels representative of a respective set ofobjects present in the respective portion of the 3D point cloud data andii) a respective set of 2D labels representative of a respective set ofobjects present in the respective portion of the 2D image data.

In an example embodiment, the method further includes determining, foreach synchronized frame, a respective set of similarity scores, whereeach similarity score in the respective set of similarity scorescorresponds to a respective 3D label and a respective 2D label andrepresents a quantitative value indicative of a likelihood that therespective 3D label and the respective 2D label represent a same objectin the synchronized frame.

In an example embodiment, the method further includes generating, foreach synchronized frame, a respective scoring matrix comprising therespective set of similarity scores and determining that a particular 3Dlabel and a particular 2D label form a matching label pairrepresentative of a same particular object in the synchronized frame.

In an example embodiment, determining that the particular 3D label andthe particular 2D label form the matching label pair includesdetermining that a particular similarity score between the particular 3Dlabel and the particular 2D label is greater than each other similarityscore associated with the particular 3D label and each other similarityscore associated with the particular 2D label.

In an example embodiment, the fused sensor data includes an indicationof the matching label pair.

In an example embodiment, the method further includes calibrating a setof extrinsics for the first vehicle sensor and the second vehiclesensor, the calibrated set of extrinsics including rotational andtranslational transformation data between the first vehicle sensor andthe second vehicle sensor.

In an example embodiment, performing the frame synchronization includesperforming the frame synchronization based at least in part on thecalibrated set of extrinsics.

In an example embodiment, a system for fusing sensor data via metadataassociation is disclosed. The system includes at least one processor andat least one memory storing computer-executable instructions. The atleast one processor is configured to access the at least one memory andexecute the computer-executable instructions to perform a set ofoperations including capturing first sensor data using a first vehiclesensor and second sensor data using a second vehicle sensor andassociating first metadata with the first sensor data to obtain labeledfirst sensor data and second metadata with the second sensor data toobtain labeled second sensor data. The set of operations furtherincludes performing a frame synchronization between the labeled firstsensor data and the labeled second sensor data to obtain a set offrames, where each frame includes a respective portion of the labeledfirst sensor data and a corresponding respective portion of the labeledsecond sensor data. The set of operations additionally includesexecuting, for each frame in the set of frames, a metadata associationalgorithm to the labeled first sensor data and the labeled second sensordata and generating, based at least in part on an output of executingthe metadata association algorithm, fused sensor data from the labeledfirst sensor data and the labeled second sensor data.

The above-described system is further configured to perform any of theoperations/functions and may include any of the additionalfeatures/aspects of example embodiments of the invention described abovein relation to example computer-implemented methods of the invention.

In an example embodiment, a computer program product for fusing sensordata via metadata association is disclosed. The computer program productincludes a non-transitory computer readable medium storingcomputer-executable program instructions that, when executed by aprocessing circuit, cause a method to be performed. In an exampleembodiment, the method includes capturing first sensor data using afirst vehicle sensor and second sensor data using a second vehiclesensor and associating first metadata with the first sensor data toobtain labeled first sensor data and second metadata with the secondsensor data to obtain labeled second sensor data. The method furtherincludes performing a frame synchronization between the labeled firstsensor data and the labeled second sensor data to obtain a set offrames, where each frame includes a respective portion of the labeledfirst sensor data and a corresponding respective portion of the labeledsecond sensor data. The method additionally includes executing, for eachframe in the set of frames, a metadata association algorithm to thelabeled first sensor data and the labeled second sensor data andgenerating, based at least in part on an output of executing themetadata association algorithm, fused sensor data from the labeled firstsensor data and the labeled second sensor data.

The above-described computer program product is further configured toperform any of the operations/functions and may include any of theadditional features/aspects of example embodiments of the inventiondescribed above in relation to example computer-implemented methods ofthe invention.

These and other features of the systems, methods, and non-transitorycomputer readable media disclosed herein, as well as the methods ofoperation and functions of the related elements of structure and thecombination of parts and economies of manufacture, will become moreapparent upon consideration of the following description and theappended claims with reference to the accompanying drawings, all ofwhich form a part of this specification, wherein like reference numeralsdesignate corresponding parts in the various figures. It is to beexpressly understood, however, that the drawings are for purposes ofillustration and description only and are not intended as a definitionof the limits of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of various embodiments of the present technology areset forth with particularity in the appended claims. A betterunderstanding of the features and advantages of the technology will beobtained by reference to the following detailed description that setsforth illustrative embodiments, in which the principles of the inventionare utilized, and the accompanying drawings of which:

FIG. 1 is an aerial view of a sensor assembly that includes a LiDARsensor and a plurality of cameras in accordance with an exampleembodiment of the invention.

FIG. 2 is a hybrid data flow and block diagram illustrating fusion oflabeled two-dimensional (2D) and three-dimensional (3D) sensor data inaccordance with an example embodiment of the invention.

FIG. 3 is a process flow diagram of an illustrative method for sensordata fusion through metadata association in accordance with an exampleembodiment of the invention.

FIG. 4 is a process flow diagram of an illustrative method for fusinglabeled two-dimensional (2D) and three-dimensional (3D) sensor data inaccordance with an example embodiment of the invention.

FIG. 5 is a process flow diagram of an illustrative method for executinga sensor data label association algorithm in accordance with an exampleembodiment of the invention.

FIG. 6 is a schematic block diagram illustrating an example networkedarchitecture configured to implement example embodiments of theinvention.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various embodiments of theinvention. However, one skilled in the art will understand that theinvention may be practiced without these details. Moreover, whilevarious embodiments of the invention are disclosed herein, manyadaptations and modifications may be made within the scope of theinvention in accordance with the common general knowledge of thoseskilled in this art. Such modifications include the substitution ofknown equivalents for any aspect of the invention in order to achievethe same result in substantially the same way.

Unless the context requires otherwise, throughout the presentspecification and claims, the word “comprise” and variations thereof,such as, “comprises” and “comprising” are to be construed in an open,inclusive sense, that is as “including, but not limited to.” Recitationof numeric ranges of values throughout the specification is intended toserve as a shorthand notation of referring individually to each separatevalue falling within the range inclusive of the values defining therange, and each separate value is incorporated in the specification asit were individually recited herein. Additionally, the singular forms“a,” “an” and “the” include plural referents unless the context clearlydictates otherwise. The phrases “at least one of,” “at least oneselected from the group of,” or “at least one selected from the groupconsisting of,” and the like are to be interpreted in the disjunctive(e.g., not to be interpreted as at least one of A and at least one ofB).

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present invention. Thus, the appearances of thephrases “in one embodiment” or “in an embodiment” in various placesthroughout this specification are not necessarily all referring to thesame embodiment, but may be in some instances. Furthermore, theparticular features, structures, or characteristics may be combined inany suitable manner in one or more embodiments.

In general, a vehicle (e.g., an autonomous vehicle, a driverlessvehicle, etc.) can have a myriad of sensors onboard the vehicle. Suchsensors can be disposed on an exterior or in an interior of a vehicleand can include, without limitation, LiDAR sensors, radars, cameras, GPSreceivers, sonar-based sensors, ultrasonic sensors, IMUs,accelerometers, gyroscopes, magnetometers, FIR sensors, and so forth.Such sensors play a central role in the functioning and operation of anautonomous or driverless vehicle. For example, LiDARs can be utilized todetect objects (e.g., other vehicles, road signs, pedestrians,buildings, etc.) in an environment around a vehicle. LiDARs can also beutilized to determine relative distances between objects in theenvironment and between objects and the vehicle. As another non-limitingexample, radars can be utilized in connection with collision avoidance,adaptive cruise control, blind spot detection, assisted parking, andother vehicle applications. As yet another non-limiting example, camerascan be utilized to capture images of an environment and object detectionprocessing can be executed on the captured images to recognize,interpret, and/or identify objects in the images and/or visual cues ofthe objects. Cameras and other optical sensors can capture image datausing charge coupled devices (CCDs), complementary metal oxidesemiconductors (CMOS), or similar elements. Data collected from thesesensors can be processed and used, as inputs, to algorithms configuredto make various autonomous driving decisions including decisionsrelating to when and how much to accelerate, decelerate, changedirection, or the like.

Various pre-processing may be performed on sensor data captured bydifferent types of sensors before the sensor data is provided as inputto algorithms, calculations, or the like that are executed/performed inconnection with operations relating to autonomous vehicle operation, forexample. For instance, sensor data captured by various sensors may beused as a training dataset to train one or more machine learningmodels/classifiers that may be used in connection with a range of taskssuch as object detection, instance segmentation, 3D regression, vehiclenavigation, and the like. In example scenarios, various metadata mayneed to be associated with the sensor data prior to the sensor databecoming usable as a training dataset. Such metadata may include, forexample, labels that are assigned to/associated with the sensor data.The labels may include indicia that identify the locations of variousobjects present in the sensor data. Additionally, the labels mayidentify object types of the objects. In example scenarios, the labelsmay be manually associated with the sensor data to form the trainingdataset.

It is often the case that metadata is associated separately and atdifferent times with sensor data from different sensors. For example, 3Dpoint cloud data captured by a LiDAR sensor may be labeled separatelythan 2D image data captured by a camera. As a result, correspondencebetween the labeled 3D point cloud data and the labeled 2D image datamay not be established. More specifically, a correspondence betweenlabeled objects in the 3D data and labeled objects in the 2D data maynot exist. This, in turn, creates a technical problem in fusing the 3Dand 2D data to obtain a training dataset that can be used to train adeep learning model/classifier. In particular, conventional techniquesfor separately associating metadata (e.g., labels) with sensor data suchas 2D and 3D sensor data present a technical problem relating to fusingthe data because it is not known which labels in the 2D data and whichlabels in the 3D data correspond to the same objects.

Various embodiments of the invention overcome technical problemsspecifically arising in the realm of computer-based technology, and morespecifically, in the realm of autonomous vehicle technology. Inparticular, example embodiments of the invention provide technicalsolutions to the above-described technical problem in the form ofsystems, methods, non-transitory computer-readable media, techniques,and methodologies for fusing different types of sensor data (e.g., 2Dimage data and 3D point cloud data) by associating respective metadata(e.g., labels) associated with the sensor data.

In example embodiments, first sensor data may be captured by a firstvehicle sensor over some period of time. The first vehicle sensor maybe, for example, a LiDAR sensor that captures 3D point cloud datarepresentative of a sensed environment. The 3D point cloud data mayinclude a collection of points that represent various 3D shapes/objectssensed within the environment scanned by the LiDAR. Each such point inthe point cloud data may be associated with a corresponding (x, y, z)coordinate within a reference coordinate system. Similarly, over thesame period of time, second sensor data may be captured by a secondvehicle sensor. The second sensor data may be, for example, 2D imagedata captured by a camera or other type of image sensor. The 2D imagedata may include a collection of pixels, with each pixel havingcorresponding RGB values, for example.

In example embodiments, respective metadata may be associated with eachof the 3D point cloud data and the 2D image data. In exampleembodiments, first metadata associated with the 3D point cloud data maybe, for example, labels applied to objects detected in the 3D pointcloud data. The labels may include, for example, a 3D bounding box(e.g., rectangular prism) formed around each object identified in the 3Dpoint cloud data. Similarly, in example embodiments, second metadata maybe associated with the 2D image data. The second metadata may be, forexample, labels such as 2D bounding boxes (e.g., rectangles) formedaround objects identified in the 2D image data. The labels may furtherinclude classifications of the objects in the 3D and 2D sensor data. Forinstance, each label applied to the 3D point cloud data may include a 3Dbounding box formed around a corresponding object identified in the 3Ddata as well as an indication of the type of object (e.g., a vehicle, apedestrian, a static object, etc.). Similarly, each label applied to the2D image data may include a 2D bounding box formed around acorresponding object identified in the 2D data as well as an indicationof the type of object. In some example embodiments, the object typeidentifier may be a code or the like that identifies the type of object.

As previously noted, the 2D image data and 3D point cloud data may belabeled separately from one another and potentially over the course ofdifferent time periods. As a result, a sufficient enough correspondencemay not exist between the labeled 2D image data and the labeled 3D pointcloud data to enable the 2D and 3D data to be fused and used as acombined training dataset to train a deep learning/machine learningmodel. That is, the labels assigned to the 3D data may not be linked orassociated with the labels assigned to the 2D image data, and as such,it may not be possible to determine if an object labeled in the 3D datais a same object labeled in the 2D data.

Example embodiments of the invention provide a technical solution tothis technical problem by performing a frame synchronization of thelabeled 3D point cloud data and the labeled 2D image data to obtain aset of synchronized frames, where each synchronized frame corresponds toa frame of the 2D image data and a corresponding frame of the 3D pointcloud data when the camera and the LiDAR are capturing the same portionof a scene. Because the LiDAR and the camera occupy different positionson the vehicle and each have a different field-of-view (FOV) and adifferent data frame capture rate, a set of extrinsics may be calibratedfor the LiDAR and the camera and used along with timing data from theLiDAR and camera to perform the frame synchronization. The calibratedset of extrinsics may include rotational and translational informationthat, along with timing data, allows for data frames captured by theLiDAR to be aligned with image frames captured by the camera to ensurethat a LiDAR data frame and an image frame, when synchronized,correspond to data captured from the same (or substantially the same)portion of a scene over the same (or substantially the same) period oftime.

After the frame synchronization is performed, a set of synchronizedframes is obtained, where each synchronized frame corresponds to aparticular image frame of the 2D image data and a particular data frameof the 3D point cloud data and is associated with a set of labelsassigned to the particular image frame and a set of labels assigned tothe particular data frame of the 3D point cloud data. Then, a metadataassociation algorithm, or more specifically, a label associationalgorithm may be executed on the set of synchronized frames on aframe-by-frame basis to fuse the 2D image data and the 3D point clouddata.

In example embodiments, executing the label association algorithm for aparticular synchronized frame, that is, a particular data frame capturedby the LiDAR and a particular 2D image frame synchronized with theparticular LiDAR frame, may include determining a set of similarityscores between the 2D labels associated with the particular 2D imageframe and the 3D labels associated with the particular LiDAR frame. Morespecifically, a respective similarity score may be determined betweeneach pair of 2D label and 3D label. The similarity score may be based onvarious parameters including, without limitation, the object type of theobject corresponding to the label, the shape of the labeled object, thelocation of the labeled object within the frame, and so forth. Then, asimilarity score matrix may be generated that contains the set ofdetermined similarity scores. In some example embodiments, matchingpairs of 2D and 3D labels may be identified by selecting, for each 2Dlabel, the 3D label with which the 2D label has the highest similarityscore or vice versa. In some example embodiments, a matching pair mayonly be identified if the corresponding similarity score of the matchingpair satisfies (e.g., meets or exceeds) a threshold score. Thus, in someexample embodiments, one of more 2D labels may remain unmatched if thereis no 3D label with which the 2D label has a similarity score thatsatisfies the threshold score. The converse is true as well. That is,one or more 3D labels may remain unmatched if there is no 2D label withwhich the 3D label has a similarity score that satisfies the thresholdscore.

In example embodiments, after the label association algorithm isexecuted and matching label pairs of 2D and 3D labels are identified,the 2D and 3D data can be fused to form, for example, a training datasetfor training a machine learning model. Constituent members of a matchinglabel pair (e.g., a 2D label and a 3D label) may be presumed tocorrespond to a same sensed object. Thus, fused 2D and 3D sensor datathat is indicative of matching 2D and 3D labels provides a richertraining dataset for use in training a machine learning model, and thus,results in an improved learning model that is capable of performing itsdesignated classification task across a broader range of potential typesof objects and/or with improved accuracy. Thus, example embodiments ofthe invention that provide techniques and algorithms for associatingfirst metadata (e.g., labels) assigned to first sensor data captured bya first sensor with second metadata assigned to second sensor datacaptured by a second sensor of a different type than the first sensorresult in enhanced fused sensor data that improves the training of amachine learning model that relies on the fused sensor data. Thus,example embodiments of the invention that produce enhanced fused sensordata results in an improvement to a technological field, in particular,the technical field of sensor data fusion and machine learning. Thistechnological improvement provides a technical solution to the technicalproblem posed by sensor data from different sensors being labeledseparately such that there is no established correspondence between thedifferent labeled sensor data, thereby making it difficult to generatefused sensor data that can be used as a training dataset.

In example embodiments, the metadata/label association algorithmdescribed herein may be executed in real-time during operation of avehicle such as an autonomous vehicle or may executed offline on labeled2D/3D data generated by labelers tasked with labeling the 3D point clouddata and 2D image data. For instance, in some example embodiments, 3Dpoint cloud data and 2D image data may be manually labeled offline andthe metadata/label association algorithm may be executed offline on themanually labeled data.

FIG. 1 is an aerial view of a sensor assembly 108 in accordance with anexample embodiment of the invention. The sensor assembly 108 may includea variety of different types of sensors including, for example, one ormore LiDAR sensors 104 and one or more cameras 106. Although notdepicted in FIG. 1 , the sensor assembly 108 may further include othertypes of sensors such as, for example, one or more IMUs, one or more GPSreceivers, and so forth. In the example configuration depicted in FIG. 1, the LiDAR sensor 104 is centrally located on a roof of a vehicle 102and is surrounded by multiple cameras that are positionedcircumferentially around the LiDAR sensor 104. In example embodiments,the LiDAR sensor 104 may periodically rotate through a scan path duringwhich the LiDAR 104 may illuminate objects in the scanned environmentwith periodic pulses of light and measure the differences in flighttimes and wavelengths for light that is reflected back to detect thepresence of target objects, determine distances between the vehicle 102and the target objects, determine distances between various targetobjects in the sensed environment, and the like. More specifically, theLiDAR 104 may be configured to generate digital 3D representations oftargets that were illuminated by the light pulses based on the measureddifferences in flight times and wavelengths for light that is reflected.More specifically, a LiDAR sensor may generate a 3D point cloud (a setof data points in space) representative of a target object that it hasilluminated with light during its scan path. The LiDAR 104 may exhibit ahorizontal scan path and/or a vertical scan path.

In example embodiments, as the LiDAR 104 travels through its scan path,it may become aligned with each camera 106 of the sensor assembly at arespective particular point in time. In order to perform the data framesynchronization described herein between the 3D point cloud datacaptured by the LiDAR 104 and the 2D image data captured by the cameras106, a calibrated set of extrinsics may be determined. In addition,timing data received from the LiDAR 104 and the cameras 106 (e.g.,shutter times/speeds of the cameras 106, timestamps for LiDAR scans,etc.) may be utilized. The set of extrinsics may provide variousrotational and translational information that can be used in conjunctionwith the timing data to determine relative positioning of the LiDAR 104with respect to any given camera 106 at any point in time, therebyallowing for each LiDAR data frame to be matched with correspondingimage frame capturing a same portion of the sensed environment at a samepoint in time, and as a result, syncing the LiDAR data frame with the 2Dcamera image frame. In addition, in accordance with example embodimentsof the disclosure, the calibrated set of extrinsics may be used inconnection with execution of the label association algorithm todetermine the location of 2D labels in 3D space and the location of 3Dlabels in 2D space.

FIG. 2 is a hybrid data flow and block diagram illustrating fusion oflabeled two-dimensional (2D) and three-dimensional (3D) sensor data inaccordance with an example embodiment of the invention. FIG. 3 is aprocess flow diagram of an illustrative method 300 for sensor datafusion through metadata association in accordance with an exampleembodiment of the invention. FIG. 4 is a process flow diagram of anillustrative method 400 for fusing labeled two-dimensional (2D) andthree-dimensional (3D) sensor data in accordance with an exampleembodiment of the invention. FIG. 5 is a process flow diagram 500 of anillustrative method for executing a sensor data label associationalgorithm in accordance with an example embodiment of the invention.Each of FIGS. 3-5 will be described in conjunction with FIG. 2hereinafter.

Each operation of any of the methods 300-500 can be performed by one ormore of the engines/program modules depicted in FIG. 2 or in FIG. 6 ,whose operation will be described in more detail hereinafter. Theseengines/program modules can be implemented in any combination ofhardware, software, and/or firmware. In certain example embodiments, oneor more of these engines/program modules can be implemented, at least inpart, as software and/or firmware modules that includecomputer-executable instructions that when executed by a processingcircuit cause one or more operations to be performed. In exampleembodiments, these engines/program modules may be customizedcomputer-executable logic implemented within a customized computingmachine such as a customized FPGA or ASIC. A system or device describedherein as being configured to implement example embodiments of theinvention can include one or more processing circuits, each of which caninclude one or more processing units or cores. Computer-executableinstructions can include computer-executable program code that whenexecuted by a processing core can cause input data contained in orreferenced by the computer-executable program code to be accessed andprocessed by the processing core to yield output data.

Referring first to FIG. 2 , a vehicle 202 is depicted. The vehicle 202may be any suitable type of vehicle including, without limitation, acar; a truck; a two-wheeled vehicle such as a motorcycle, moped,scooter, or the like; a vehicle with more than two axles (e.g., atractor trailer); and so forth. The vehicle 202 include various on-boardvehicle sensors such as a LiDAR 204 and one or more cameras 206. Inexample embodiments, the vehicle 202 may be the vehicle 102 and theLiDAR 204 and camera 206 may form part of the sensor assembly 108depicted in FIG. 1 .

Referring now to FIG. 3 in conjunction with FIG. 2 , at block 302 of theexample method 300, a sensor extrinsics calibration engine 224 may beexecuted to calibrate extrinsics for a first vehicle sensor (e.g., theLiDAR 204) and a second vehicle sensor (e.g., the camera 206). Thesensor extrinsics calibration engine 224 may generate sensor extrinsicsdata 226. The data 226 may include rotational and translationalinformation (e.g., rotational and translational matrices) thatdetermines a relative position of the first sensor with respect to thesecond sensor. The sensor extrinsics data 226 can be used, along withtiming data from the sensors, to perform a data synchronization between3D data frames captured by the LiDAR 204 and 2D image frames captured bythe camera 206 as well as to locate 2D labeled objects in 3D space andvice versa, as will be described in further detail later in thisdisclosure.

At block 304 of the method 300, first sensor data may be captured by thefirst vehicle sensor and second sensor data may be captured by thesecond vehicle sensor. The first sensor data may be 3D point cloud data212 captured by the LiDAR 204 during its scan path. The second sensordata may be 2D image data 214 captured by the camera 206. Both the 3Ddata 212 and the 2D data 214 may correspond to various portions of ascene 210 sensed by the sensors, which may include various differenttypes of objects.

At block 306 of the method 300, timing data 236 may be received from thefirst vehicle sensor and the second vehicle sensor. For instance, timingdata may be received from the LiDAR 204 in the form of timestampsassociated with vertical and/or horizontal scans performed by the LiDAR204 as it travels along the scan path 208. Timing data received from thecamera 206 may include, for example, shutter times/speeds indicative ofthe amount of time (e.g., number of milliseconds) that it takes for thecamera 206 to capture an image.

At block 308 of the method 300, a metadata assignment engine 216 may beexecuted to associate first metadata with the first sensor data andsecond metadata with the second sensor data. Associating the firstmetadata with the first sensor data may include assigning labels to the3D point cloud data 212 to obtained labeled 3D data 218. Associating thesecond metadata with the second sensor data may include assigning labelsto the 2D image data 214 to obtain labeled 2D data 220. As will bedescribed in more detail later in this disclosure, the labels assignedto the 3D data 212 may include 3D bounding boxes formed around objectsidentified in the 3D data 212, object type classifications/identifiersassigned to the identified objects, and so forth. Similarly, the labelsassigned to the 2D image data 214 may include 2D bounding boxes formedaround objects identified in the 2D data 214, object typeclassifications/identifiers assigned to the identified objects, and soforth.

At block 310 of the method 300, a sensor data frame synchronizationengine 222 may be executed to perform a frame synchronization betweenthe first sensor data (e.g., the 3D data 212 or the labeled 3D data 218)and the second sensor data (e.g., the 2D data 214 or the labeled 2Dimage data 220) using the sensor extrinsics data 226 in conjunction withthe timing data 236 to obtain frame synced data 228. While the framesynchronization performed at block 310 is illustratively depicted inFIGS. 2 and 3 , for example, as occurring after the metadata assignmentat block 308, this may not be case. That is, in example embodiments,labeling of the 2D and 3D data is not necessary to perform the framesynchronization, and thus, the frame synchronization may occur prior to,after, and/or at least partially concurrently with the metadataassignment (e.g., the labeling of the 2D/3D data). For ease ofexplanation, however, the frame synchronization will be describedhereinafter as occurring after the labeling of the 2D and 3D data and asbeing performed on the labeled data.

The frame synced data 228 may include a collection of synced frames inwhich each 3D data frame in the labeled 3D data 218 is synced with arespective corresponding image frame of the labeled 2D image data 220. A3D data frame synced with a 2D image frame may indicate that the framescorrespond to sensor data captured from a same or substantially a sameportion of the scene 210 at the same or substantially the same period oftime, and thus, that the LiDAR 204 that generated the 3D data in the 3Ddata frame and the camera 206 that generated the image data in the 2Dimage frame were positionally aligned at that period of time. It shouldbe appreciated that a 3D data frame of the labeled 3D point cloud data218 may be frame synced with multiple 2D image frames. Each synced framein the frame synced data 228 may be associated with a set of 2D labelsassociated with a corresponding 2D image frame and a set of 3D labelsassociated with a 3D data frame synced with the 2D image frame.

Finally, at block 312 of the method 300, a metadata association engine232 may execute a metadata association algorithm on the frame synceddata 228 on a frame-by-frame basis to fuse the labeled first sensor data(e.g., the labeled 3D data 218) and the labeled second sensor data(e.g., the labeled 2D data 220) and generate fused sensor data 234.Executing the metadata association algorithm for a given synced frame ofthe frame synced data 228 may include determining one or more matching2D/3D label pairs, where each matching 2D/3D label pair corresponds to asame object in the sensed environment. Execution of the metadataassociated algorithm will be described in more detail later in thisdisclosure.

The example method 400 provides a more specific implementation ofmetadata association techniques described herein with respect to 3Dpoint cloud data captured by the LiDAR 204 and 2D image data captured bythe camera 206. Referring now to FIG. 4 in conjunction with FIG. 2 , atblock 402 of the method 400, a relative position between the LiDAR 204and the camera 206 may be determined. In some example embodiments, therelative positioning of the LiDAR 204 with respect to the camera 206 maybe determined by executing the sensor extrinsics calibration engine 224to determine the sensor extrinsics data 226, which may includetranslational and rotational data that can be used to determine arelative alignment/positioning between the LiDAR 204 and the camera 204at any given point in time.

At block 404 of the method 400, the 3D point cloud data 212corresponding to various portions of the scene 210 may be captured usingthe LiDAR 204. The 3D point cloud data 212 may include data captured bythe LiDAR 206 as it traverses a scan path 208, which may be a verticalscan path and/or a horizontal scan path. It should be appreciated thatthe LiDAR 204 may traverse a scan path that includes a 360 degreerotation, and that the scan path 208 may represent only a portion of thetotal scan path of the LiDAR 204, and thus, the scene 210 may representonly a portion of the total environment sensed by the LiDAR 204.

The 3D point cloud data 212 may include a collection of points thatrepresent various 3D shapes/objects sensed within the environmentscanned by the LiDAR 204 including the scene 210. Each such point in thepoint cloud data 212 may be associated with a corresponding (x, y, z)coordinate within a reference coordinate system. In example embodiments,the reference coordinate system may be centered at a location of theLiDAR 204 with respect to the vehicle 202, at an origin of a globalcoordinate system, or the like.

At block 406 of the method 400, the 2D image data 214 corresponding tovarious portions of the scene 210 may be captured using the camera 206.It should be appreciated that multiple cameras 206 forming part of thesensor assembly 108 (FIG. 1 ), for example, may capture the 2D imagedata 214. The 2D image data 214 may include a collection of pixels, witheach pixel having corresponding RGB values, for example.

At block 408 of the method 400, respective timing data 236 may bereceived from each of the LiDAR sensor 204 and the camera 206. Thetiming data received from the LiDAR may include timestamps associatedwith vertical and/or horizontal scans performed by the LiDAR 204 as ittravels along the scan path 208. For example, a set of timestamps may bereceived for each scan of the LiDAR 204 including a first timestampindicative a start time of the scan and a second timestamp indicating anend time of the scan. Timing data received from the camera 206 mayinclude, for example, shutter times/speeds indicative of the amount oftime (e.g., number of milliseconds) that it takes for the camera 206 tocapture an image.

At block 410 of the method 400, the metadata assignment engine 216 maybe executed to assign labels to the 3D point cloud data 212 to obtainlabeled 3D point cloud data 218. More specifically, the metadataassignment engine 216 may be executed to assign labels to objectsidentified in the 3D point cloud data 212. Further, at block 412 of themethod 400, the metadata assignment engine 216 may be executed to assignlabels to the 2D image data 214 to obtain labeled 2D image data 220.More specifically, the metadata assignment engine 216 may be executed toassign labels to objects identified in the 2D data 220.

In example embodiments, labels applied to objects detected in the 3Dpoint cloud data 212 may include, for example, a 3D bounding box (e.g.,rectangular prism) formed around each object identified in the 3D pointcloud data 212. In example embodiments, each such 3D bounding box may bedefined by a set of coordinates within a coordinate system. Forinstance, if the bounding box is a rectangular prism, the bounding boxmay be defined by a minimum of two coordinates: a first coordinaterepresenting a particular corner of the rectangular prism and a secondcoordinate that opposes the first coordinate along a length, width, anddepth of the rectangular prism. While a greater number of coordinatesmay be specified, only these 2 particular coordinates are needed tofully define a rectangular prism bounding box within a coordinatesystem. In other example embodiments, other 3D structures (e.g., aspherical structure) may be used as a 3D bounding box.

Similarly, in example embodiments, the labels associated with the 2Dimage data 214 may include 2D bounding boxes such as rectangles formedaround objects identified in the 2D image data 214. In exampleembodiments, each such 2D bounding box may be defined by a set ofcoordinates within a coordinate system. For instance, if the boundingbox is a rectangle, the bounding box may be defined by a minimum of twocoordinates: a first coordinate representing a particular corner of therectangle (e.g., an upper left corner of the rectangle) and a secondcoordinate representing an opposing corner of the rectangle (e.g., alower right corner of the rectangle).

The 3D labels and/or the 2D labels may further include classificationsof the objects in the respective 3D data 212 and 2D data 214. Forinstance, each label applied to the 3D point cloud data 212 may includea 3D bounding box formed around a corresponding object identified in the3D data 212 as well as an indication of the type of object (e.g., avehicle, a pedestrian, a static object, etc.). Similarly, each labelapplied to the 2D image data 214 may include a 2D bounding box formedaround a corresponding object identified in the 2D data 214 as well asan indication of the type of object. In some example embodiments, theobject type identifier may be a code or the like that identifies thetype of object.

At block 414 of the method 400, sensor data frame synchronization engine222 may be executed to perform, based on the sensor extrinsics data 226(e.g., the relative position between the LiDAR 204 and the camera 206)and the timing data 236, a frame synchronization between the labeled 3Dpoint cloud data 218 and the labeled 2D image data 220 to obtain framesynced data 228. The calibrated sensor extrinsics data 226 may includerotational and translational information that allows for data framescaptured by the LiDAR 204 to be aligned with image frames captured bythe camera 206 to ensure that a LiDAR data frame and an image frame,when synchronized, correspond to data captured from the same (orsubstantially the same) portion of the scene 210 over the same (orsubstantially the same) period of time. In example embodiments, eachsynced frame in the frame synced data 228 may be associated with a setof 2D labels assigned to a corresponding 2D image data frame in thelabeled 2D image data 220 and a set of 3D labels assigned to acorresponding 3D LiDAR data frame in the labeled 3D data 218 that hasbeen synced with the 2D image frame. As previously noted, the framesynchronization may alternatively occur prior to and/or at leastpartially concurrently with the assignment of the 2D and 3D labels tothe 2D image data 214 and 3D data 212, respectively, in which case, theframe synchronization may be performed between at least partiallyunlabeled 3D point cloud data 212 and at least partially unlabeled 2Dimage data 214 to obtain the frame synced data 228. In such exampleembodiments, at least some of the 2D and/or 3D labels may be assigned tothe 2D data 214 and 3D data 212, respectively, after the framesynchronization is performed. That is, in some example embodiments, atleast some of the 2D and/or 3D labels may be assigned to the framesynced data 228.

At block 416 of the method 400, the metadata association engine 232 mayexecute a label association algorithm on the frame synced data 228 on aper frame basis to determine associations between the 2D labels and the3D labels and generate corresponding fused 2D/3D sensor data 234indicative of the determined associations. FIG. 5 depicts an examplemethod 500 representing a particular implementation of the labelassociation algorithm executed at block 414 of the method 400 for aparticular synced data frame.

Referring now to FIG. 5 , at block 502 of the method 500, the metadataassociation engine 232 may execute the label association algorithm todetermine respective similarity scores between 2D labels (e.g., 2Dbounding boxes) associated with the synced data frame and 3D labels(e.g., 3D bounding boxes) associated with the synced data frame. Thatis, for a particular synchronized frame, e.g., a particular 3D dataframe captured by the LiDAR 204 and a particular 2D image frame capturedby the camera 206 and synchronized with the particular LiDAR frame, themetadata association engine 232 may determine a set of similarity scoresbetween the 2D labels associated with the particular 2D image frame andthe 3D labels associated with the particular LiDAR frame. Morespecifically, a respective similarity score may be determined betweeneach pair of 2D label and 3D label. In example embodiments, eachsimilarity score may be a quantitative value indicative of a likelihoodthat a 2D bounding box and a 3D bounding box to which the similarityscore corresponds represents the same object.

In example embodiments, the similarity score may be based on variousparameters including, without limitation, the object type of the objectcorresponding to the label, the shape of the labeled object, thelocation of the labeled object within the frame, and so forth. In someexample embodiments, the sensor extrinsics data 226 including therotational and translational data included therein may be used to locatea 2D object labeled in the 2D image frame in the 3D space of the 3DLiDAR data frame. This may include extrapolating the 2D object to acorresponding 3D object in the 3D space. Similarly, the sensorextrinsics data 226 may be used to locate a 3D object labeled in the 3DLiDAR data frame in the 2D space of the 2D image frame.

Optionally, at block 504 of the method 500, the metadata associationengine 232 may perform one or more optimizations to enhance theperformance of the label association algorithm. An example optimizationmay be to cease determining additional similarity scores for aparticular 2D label or a particular 3D label after some threshold numberof previously determined similarity scores are below a threshold scorevalue. Another example optimization may be to cease determiningadditional similarity scores for a particular 2D label or a particular3D label after a similarity score above a threshold score value isdetermined. It should be appreciated that the above examples ofpotential optimizations are merely illustrative and not exhaustive.

Then, at block 506 of the method 500, a similarity score matrix may begenerated that contains the set of determined similarity scores. Atblock 508 of the method 500, matching pairs of 2D and 3D labels may beidentified by selecting, for each 2D label, the 3D label with which the2D label has the highest similarity score or vice versa. In some exampleembodiments, a matching pair may only be identified if the correspondingsimilarity score of the matching pair satisfies (e.g., meets or exceeds)a threshold score. Thus, in some example embodiments, one of more 2Dlabels may remain unmatched if there is no 3D label with which the 2Dlabel has a similarity score that satisfies the threshold score. Theconverse is true as well. That is, one or more 3D labels may remainunmatched if there is no 2D label with which the 3D label has asimilarity score that satisfies the threshold score.

In example embodiments, after the label association algorithm isexecuted and matching label pairs of 2D and 3D labels are identified,the 2D and 3D data can be fused at block 510 of the method 500 to formfused sensor data 234 that can be used, for example, as a trainingdataset for training a machine learning model. Constituent members of amatching label pair (e.g., a 2D label and a 3D label) may be presumed tocorrespond to a same sensed object. Thus, fused 2D and 3D sensor data234 that is indicative of matching 2D and 3D labels provides a richertraining dataset for use in training a machine learning model, and thus,results in an improved learning model that is capable of performing itsdesignated classification task across a broader range of potential typesof objects and/or with improved accuracy.

Hardware Implementation

FIG. 6 is a schematic block diagram illustrating an example networkedarchitecture 600 configured to implement example embodiments of theinvention. The networked architecture 600 can include one or morespecial-purpose computing devices 602 communicatively coupled via one ormore networks 606 to various sensors 604. The sensors 604 may includeany of the example types of on-board vehicle sensors previouslydescribed including, without limitation, LiDAR sensors, radars, cameras,GPS receivers, sonar-based sensors, ultrasonic sensors, IMUs,accelerometers, gyroscopes, magnetometers, FIR sensors, and so forth. Inexample embodiments, the sensors 604 may include on-board sensorsprovided on an exterior or in an interior of a vehicle such as anautonomous vehicle. The special-purpose computing device(s) 602 mayinclude devices that are integrated with a vehicle and may receivesensor data from the sensors 604 via a local network connection (e.g.,WiFi, Bluetooth, Dedicated Short Range Communication (DSRC), or thelike). In other example embodiments, the special-purpose computingdevice(s) 602 may be provided remotely from a vehicle and may receivethe sensor data from the sensors 604 via one or more long-rangenetworks.

The special-purpose computing device(s) 602 may be hard-wired to performthe techniques; may include circuitry or digital electronic devices suchas one or more ASICs or FPGAs that are persistently programmed toperform the techniques; and/or may include one or more hardwareprocessors programmed to perform the techniques pursuant to programinstructions in firmware, memory, other storage, or a combinationthereof. The special-purpose computing device(s) 602 may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing device(s) 602may be desktop computer systems, server computer systems, portablecomputer systems, handheld devices, networking devices or any otherdevice or combination of devices that incorporate hard-wired and/orprogrammed logic to implement the techniques.

The special-purpose computing device(s) may be generally controlled andcoordinated by operating system software 620, such as iOS, Android,Chrome OS, Windows XP, Windows Vista, Windows 4, Windows 8, WindowsServer, Windows CE, Unix, Linux, SunOS, Solaris, iOS, Blackberry OS,VxWorks, or other compatible operating systems. In other embodiments,the computing device(s) 602 may be controlled by a proprietary operatingsystem. The operating system software 620 may control and schedulecomputer processes for execution; perform memory management; providefile system, networking, and I/O services; and provide user interfacefunctionality, such as a graphical user interface (“GUI”).

While the computing device(s) 602 and/or the sensors 604 may bedescribed herein in the singular, it should be appreciated that multipleinstances of any such component can be provided and functionalitydescribed in connection any particular component can be distributedacross multiple instances of such a component. In certain exampleembodiments, functionality described herein in connection with any givencomponent of the architecture 600 can be distributed among multiplecomponents of the architecture 600. For example, at least a portion offunctionality described as being provided by a computing device 602 maybe distributed among multiple such computing devices 602.

The network(s) 606 can include, but are not limited to, any one or moredifferent types of communications networks such as, for example, cablenetworks, public networks (e.g., the Internet), private networks (e.g.,frame-relay networks), wireless networks, cellular networks, telephonenetworks (e.g., a public switched telephone network), or any othersuitable private or public packet-switched or circuit-switched networks.The network(s) 606 can have any suitable communication range associatedtherewith and can include, for example, global networks (e.g., theInternet), metropolitan area networks (MANs), wide area networks (WANs),local area networks (LANs), or personal area networks (PANs). Inaddition, the network(s) 606 can include communication links andassociated networking devices (e.g., link-layer switches, routers, etc.)for transmitting network traffic over any suitable type of mediumincluding, but not limited to, coaxial cable, twisted-pair wire (e.g.,twisted-pair copper wire), optical fiber, a hybrid fiber-coaxial (HFC)medium, a microwave medium, a radio frequency communication medium, asatellite communication medium, or any combination thereof.

In an illustrative configuration, the computing device 602 can includeone or more processors (processor(s)) 608, one or more memory devices610 (generically referred to herein as memory 610), one or moreinput/output (“I/O”) interface(s) 612, one or more network interfaces614, and data storage 618. The computing device 602 can further includeone or more buses 618 that functionally couple various components of thecomputing device 602. The data storage may store one or more engines,program modules, components, or the like including, without limitation,a virtual sensor system 624 that represents a virtual simulation of theoperation of one or more of the sensors 604. The virtual sensor system624 may, in turn, include one or more engines, program modules,components, or the like including, without limitation, a predictivemodel 626 and a training/calibration engine 628. Each of theengines/components depicted in FIG. 6 may include logic for performingany of the processes or tasks described earlier in connection withcorrespondingly named engines/components. In certain exampleembodiments, any of the depicted engines/components may be implementedin hard-wired circuitry within digital electronic devices such as one ormore ASICs or FPGAs that are persistently programmed to performcorresponding techniques.

The bus(es) 618 can include at least one of a system bus, a memory bus,an address bus, or a message bus, and can permit the exchange ofinformation (e.g., data (including computer-executable code), signaling,etc.) between various components of the computing device 602. Thebus(es) 618 can include, without limitation, a memory bus or a memorycontroller, a peripheral bus, an accelerated graphics port, and soforth. The bus(es) 618 can be associated with any suitable busarchitecture including, without limitation, an Industry StandardArchitecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA(EISA), a Video Electronics Standards Association (VESA) architecture,an Accelerated Graphics Port (AGP) architecture, a Peripheral ComponentInterconnects (PCI) architecture, a PCI-Express architecture, a PersonalComputer Memory Card International Association (PCMCIA) architecture, aUniversal Serial Bus (USB) architecture, and so forth.

The memory 610 can include volatile memory (memory that maintains itsstate when supplied with power) such as random access memory (RAM)and/or non-volatile memory (memory that maintains its state even whennot supplied with power) such as read-only memory (ROM), flash memory,ferroelectric RAM (FRAM), and so forth. Persistent data storage, as thatterm is used herein, can include non-volatile memory. In certain exampleembodiments, volatile memory can enable faster read/write access thannon-volatile memory. However, in certain other example embodiments,certain types of non-volatile memory (e.g., FRAM) can enable fasterread/write access than certain types of volatile memory.

In various implementations, the memory 610 can include multipledifferent types of memory such as various types of static random accessmemory (SRAM), various types of dynamic random access memory (DRAM),various types of unalterable ROM, and/or writeable variants of ROM suchas electrically erasable programmable read-only memory (EEPROM), flashmemory, and so forth. The memory 610 can include main memory as well asvarious forms of cache memory such as instruction cache(s), datacache(s), translation lookaside buffer(s) (TLBs), and so forth. Further,cache memory such as a data cache can be a multi-level cache organizedas a hierarchy of one or more cache levels (L1, L2, etc.). In exampleembodiments, the memory 610 may include the data storage 106(1)-106(P)and/or the data storage 120 depicted in FIG. 1 . Alternatively, the datastorage 106(1)-106(P) may be hard disk storage forming part of the datastorage 618 and/or the data storage 120 may be a form of RAM or cachememory that is provided as part of the FOV semantics computing machine624 itself.

The data storage 618 can include removable storage and/or non-removablestorage including, but not limited to, magnetic storage, optical diskstorage, and/or tape storage. The data storage 618 can providenon-volatile storage of computer-executable instructions and other data.The memory 610 and the data storage 618, removable and/or non-removable,are examples of computer-readable storage media (CRSM) as that term isused herein. The data storage 618 can store computer-executable code,instructions, or the like that can be loadable into the memory 610 andexecutable by the processor(s) 608 to cause the processor(s) 608 toperform or initiate various operations. The data storage 618 canadditionally store data that can be copied to memory 610 for use by theprocessor(s) 608 during the execution of the computer-executableinstructions. Moreover, output data generated as a result of executionof the computer-executable instructions by the processor(s) 608 can bestored initially in memory 610 and can ultimately be copied to datastorage 618 for non-volatile storage.

More specifically, the data storage 618 can store one or more operatingsystems (O/S) 620 and one or more database management systems (DBMS) 622configured to access the memory 610 and/or one or more externaldatastore(s) (not depicted) potentially via one or more of the networks606. In addition, the data storage 618 may further store one or moreprogram modules, applications, engines, computer-executable code,scripts, or the like. For instance, any of the engines/componentsdepicted in FIG. 6 may be implemented as software and/or firmware thatincludes computer-executable instructions (e.g., computer-executableprogram code) loadable into the memory 610 for execution by one or moreof the processor(s) 608 to perform any of the techniques describedherein.

Although not depicted in FIG. 6 , the data storage 618 can further storevarious types of data utilized by engines/components of the computingdevice 602. Such data may include, without limitation, sensor data,feedback data including historical sensor operational data, initialparameter data, or the like. Any data stored in the data storage 618 canbe loaded into the memory 610 for use by the processor(s) 608 inexecuting computer-executable program code. In addition, any data storedin the data storage 618 can potentially be stored in one or moreexternal datastores that are accessible via the DBMS 622 and loadableinto the memory 610 for use by the processor(s) 608 in executingcomputer-executable instructions/program code.

The processor(s) 608 can be configured to access the memory 610 andexecute computer-executable instructions/program code loaded therein.For example, the processor(s) 608 can be configured to executecomputer-executable instructions/program code of the variousengines/components of the FOV semantics computing machine 624 to causeor facilitate various operations to be performed in accordance with oneor more embodiments of the invention. The processor(s) 608 can includeany suitable processing unit capable of accepting data as input,processing the input data in accordance with stored computer-executableinstructions, and generating output data. The processor(s) 608 caninclude any type of suitable processing unit including, but not limitedto, a central processing unit, a microprocessor, a Reduced InstructionSet Computer (RISC) microprocessor, a Complex Instruction Set Computer(CISC) microprocessor, a microcontroller, an Application SpecificIntegrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), aSystem-on-a-Chip (SoC), a digital signal processor (DSP), and so forth.Further, the processor(s) 608 can have any suitable microarchitecturedesign that includes any number of constituent components such as, forexample, registers, multiplexers, arithmetic logic units, cachecontrollers for controlling read/write operations to cache memory,branch predictors, or the like. The microarchitecture design of theprocessor(s) 608 can be made capable of supporting any of a variety ofinstruction sets.

Referring now to other illustrative components depicted as being storedin the data storage 618, the O/S 620 can be loaded from the data storage618 into the memory 610 and can provide an interface between otherapplication software executing on the computing device 602 and hardwareresources of the computing device 602. More specifically, the 0/S 620can include a set of computer-executable instructions for managinghardware resources of the computing device 602 and for providing commonservices to other application programs. In certain example embodiments,the 0/S 620 can include or otherwise control execution of one or more ofthe engines/program modules stored in the data storage 618. The 0/S 620can include any operating system now known or which can be developed inthe future including, but not limited to, any server operating system,any mainframe operating system, or any other proprietary ornon-proprietary operating system.

The DBMS 622 can be loaded into the memory 610 and can supportfunctionality for accessing, retrieving, storing, and/or manipulatingdata stored in the memory 610, data stored in the data storage 618,and/or data stored in external datastore(s). The DBMS 622 can use any ofa variety of database models (e.g., relational model, object model,etc.) and can support any of a variety of query languages. The DBMS 622can access data represented in one or more data schemas and stored inany suitable data repository. Datastore(s) that may be accessible by thecomputing device 602 via the DBMS 622, can include, but are not limitedto, databases (e.g., relational, object-oriented, etc.), file systems,flat files, distributed datastores in which data is stored on more thanone node of a computer network, peer-to-peer network datastores, or thelike.

Referring now to other illustrative components of the computing device602, the input/output (I/O) interface(s) 612 can facilitate the receiptof input information by the computing device 602 from one or more I/Odevices as well as the output of information from the computing device602 to the one or more I/O devices. The I/O devices can include any of avariety of components such as a display or display screen having a touchsurface or touchscreen; an audio output device for producing sound, suchas a speaker; an audio capture device, such as a microphone; an imageand/or video capture device, such as a camera; a haptic unit; and soforth. Any of these components can be integrated into the computingdevice 602 or can be separate therefrom. The I/O devices can furtherinclude, for example, any number of peripheral devices such as datastorage devices, printing devices, and so forth.

The I/O interface(s) 612 can also include an interface for an externalperipheral device connection such as universal serial bus (USB),FireWire, Thunderbolt, Ethernet port or other connection protocol thatcan connect to one or more networks. The I/O interface(s) 612 can alsoinclude a connection to one or more antennas to connect to one or morenetworks via a wireless local area network (WLAN) (such as Wi-Fi) radio,Bluetooth, and/or a wireless network radio, such as a radio capable ofcommunication with a wireless communication network such as a Long TermEvolution (LTE) network, WiMAX network, 3G network, etc.

The computing device 602 can further include one or more networkinterfaces 614 via which the computing device 602 can communicate withany of a variety of other systems, platforms, networks, devices, and soforth. The network interface(s) 614 can enable communication, forexample, with the sensors 604 and/or one or more other devices via oneor more of the network(s) 606. In example embodiments, the networkinterface(s) 614 provide a two-way data communication coupling to one ormore network links that are connected to one or more of the network(s)606. For example, the network interface(s) 614 may include an integratedservices digital network (ISDN) card, a cable modem, a satellite modem,or a modem to provide a data communication connection to a correspondingtype of telephone line. As another non-limiting example, the networkinterface(s) 614 may include a local area network (LAN) card to providea data communication connection to a compatible LAN (or a wide areanetwork (WAN) component to communicate with a WAN). Wireless links mayalso be implemented. In any such implementation, the networkinterface(s) 614 may send and receive electrical, electromagnetic, oroptical signals that carry digital data streams representing varioustypes of information.

A network link typically provides data communication through one or morenetworks to other data devices. For example, a network link may providea connection through a local network to a host computer or to dataequipment operated by an Internet Service Provider (ISP). The ISP, inturn, may provide data communication services through the world widepacket data communication network now commonly referred to as the“Internet”. Local networks and the Internet both use electrical,electromagnetic, or optical signals that carry digital data streams. Thesignals through the various network(s) 604 and the signals on networklinks and through the network interface(s) 614, which carry the digitaldata to and from the computing device 602, are example forms oftransmission media. In example embodiments, the computing device 602 cansend messages and receive data, including program code, through thenetwork(s) 606, network links, and network interface(s) 614. Forinstance, in the Internet example, a server might transmit a requestedcode for an application program through the Internet, the ISP, a localnetwork, and a network interface 614. The received code may be executedby a processor 608 as it is received, and/or stored in the data storage618, or other non-volatile storage for later execution.

It should be appreciated that the engines depicted in FIG. 6 as part ofthe computing device 602 are merely illustrative and not exhaustive. Inparticular, functionality can be modularized in any suitable manner suchthat processing described as being supported by any particular enginecan alternatively be distributed across multiple engines, programmodules, components, or the like, or performed by a different engine,program module, component, or the like. Further, one or more depictedengines may or may not be present in certain embodiments, while in otherembodiments, additional engines not depicted can be present and cansupport at least a portion of the described functionality and/oradditional functionality. In addition, various engine(s), programmodule(s), script(s), plug-in(s), Application Programming Interface(s)(API(s)), or any other suitable computer-executable code hosted locallyon the computing device 602 and/or hosted on other computing device(s)(e.g., 602) accessible via one or more of the network(s) 602, can beprovided to support functionality provided by the engines depicted inFIG. 6 and/or additional or alternate functionality. In addition,engines that support functionality described herein can be implemented,at least partially, in hardware and/or firmware and can be executableacross any number of computing devices 602 in accordance with anysuitable computing model such as, for example, a client-server model, apeer-to-peer model, and so forth.

It should further be appreciated that the computing device 602 caninclude alternate and/or additional hardware, software, and/or firmwarecomponents beyond those described or depicted without departing from thescope of the invention. More particularly, it should be appreciated thatsoftware, firmware, and/or hardware components depicted as forming partof the computing device 602 are merely illustrative and that somecomponents may or may not be present or additional components may beprovided in various embodiments. It should further be appreciated thateach of the engines depicted and described represent, in variousembodiments, a logical partitioning of supported functionality. Thislogical partitioning is depicted for ease of explanation of thefunctionality and may or may not be representative of the structure ofsoftware, hardware, and/or firmware for implementing the functionality.

In general, the terms engine, program module, or the like, as usedherein, refer to logic embodied in hardware, firmware, and/or circuitry,or to a collection of software instructions, possibly having entry andexit points, written in a programming language, such as, for example,Java, C or C++. A software engine/module may be compiled and linked intoan executable program, installed in a dynamic link library, or may bewritten in an interpreted programming language such as, for example,BASIC, Perl, or Python. It will be appreciated that softwareengines/modules may be callable from other engines/modules or fromthemselves, and/or may be invoked in response to detected events orinterrupts. Software engines/modules configured for execution oncomputing devices may be provided on a computer readable medium, such asa compact disc, digital video disc, flash drive, magnetic disc, or anyother tangible medium, or as a digital download (and may be originallystored in a compressed or installable format that requires installation,decompression or decryption prior to execution). Such software code maybe stored, partially or fully, on a memory device of the executingcomputing device, for execution by the computing device. “Open source”software refers to source code that can be distributed as source codeand/or in compiled form, with a well-publicized and indexed means ofobtaining the source, and optionally with a license that allowsmodifications and derived works. Software instructions may be embeddedin firmware and stored, for example, on flash memory such as erasableprogrammable read-only memory (EPROM). It will be further appreciatedthat hardware modules/engines may include connected logic units, such asgates and flip-flops, and/or may be further include programmable units,such as programmable gate arrays or processors.

Example embodiments are described herein as including engines or programmodules. Such engines/program modules may constitute either softwareengines (e.g., code embodied on a machine-readable medium) or hardwareengines. A “hardware engine” is a tangible unit capable of performingcertain operations and may be configured or arranged in a certainphysical manner. In various example embodiments, one or more computersystems (e.g., a standalone computer system, a client computer system,or a server computer system) or one or more hardware engines of acomputer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) asa hardware engine that operates to perform certain operations asdescribed herein.

In some embodiments, a hardware engine may be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware engine may include dedicated circuitry or logic that ispermanently configured to perform certain operations. For example, ahardware engine may be a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an application specific integratedcircuit (ASIC). A hardware engine may also include programmable logic orcircuitry that is temporarily configured by software to perform certainoperations. For example, a hardware engine may include a general-purposeprocessor or other programmable processor configured by software, inwhich case, the configured processor becomes a specific machine uniquelytailored to perform the configured functions and no longer constitutegeneral-purpose processors. It will be appreciated that the decision toimplement a hardware engine mechanically, in dedicated and permanentlyconfigured circuitry, or in temporarily configured circuitry (e.g.,configured by software) may be driven by cost and time considerations.

Accordingly, the term “engine” or “program module” should be understoodto encompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware engines are temporarily configured (e.g., programmed),each of the hardware engines need not be configured or instantiated atany one instance in time. For example, where a hardware engine includesa general-purpose processor configured by software to become aspecial-purpose processor, the general-purpose processor may beconfigured as respectively different special-purpose processors (e.g.,comprising different hardware engines) at different times. Softwareaccordingly can configure a particular processor or processors, forexample, to constitute a particular hardware engine at a given instanceof time and to constitute a different hardware engine at a differentinstance of time.

Hardware engines can provide information to, and receive informationfrom, other hardware engines. Accordingly, the described hardwareengines may be regarded as being communicatively coupled. Where multiplehardware engines exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware engines. In embodiments inwhich multiple hardware engines are configured or instantiated atdifferent times, communications between such hardware engines may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware engines have access.For example, one hardware engine may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware engine may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware engines may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute an implementation of ahardware engine. Similarly, the methods described herein may be at leastpartially processor-implemented, with a particular processor orprocessors being an example of hardware. Moreover, the one or moreprocessors may also operate to support performance of the relevantoperations in a “cloud computing” environment or as a “software as aservice” (SaaS). For example, at least some of the operations may beperformed by a group of computers (as examples of machines includingprocessors), with these operations being accessible via a network (e.g.,the Internet) and via one or more appropriate interfaces (e.g., an API).

The performance of certain of the operations of example methodsdescribed herein may be distributed among multiple processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processors may be located ina single geographic location (e.g., within a home environment, an officeenvironment, or a server farm). In other example embodiments, theprocessors may be distributed across a number of geographic locations.

The present invention may be implemented as a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions embodied thereon for causing a processor to carryout aspects of the present invention.

The computer readable storage medium is a form of non-transitory media,as that term is used herein, and can be any tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. The computer readable storage medium, and non-transitorymedia more generally, may include non-volatile media and/or volatilemedia. A non-exhaustive list of more specific examples of a computerreadable storage medium includes the following: a portable computerdiskette such as a floppy disk or a flexible disk; a hard disk; a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), a static random access memory(SRAM), or any other memory chip or cartridge; a portable compact discread-only memory (CD-ROM); a digital versatile disk (DVD); a memorystick; a solid state drive; magnetic tape or any other magnetic datastorage medium; a mechanically encoded device such as punch-cards orraised structures in a groove having instructions recorded thereon orany physical medium with patterns of holes; any networked versions ofthe same; and any suitable combination of the foregoing.

Non-transitory media is distinct from transmission media, and thus, acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire. Non-transitory media, however, can operate inconjunction with transmission media. In particular, transmission mediamay participate in transferring information between non-transitorymedia. For example, transmission media can include coaxial cables,copper wire, and/or fiber optics, including the wires that include atleast some of the bus(es) 602. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network(LAN), a wide area network (WAN), and/or a wireless network. The networkmay include copper transmission cables, optical transmission fibers,wireless transmission, routers, firewalls, switches, gateway computersand/or edge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a LAN or a WAN, or the connection may be madeto an external computer (for example, through the Internet using anInternet Service Provider (ISP)). In some embodiments, electroniccircuitry including, for example, programmable logic circuitry, FPGAs,or programmable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions. These computer readable programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks. These computer readable program instructions may also be storedin a computer readable storage medium that can direct a computer, aprogrammable data processing apparatus, and/or other devices to functionin a particular manner, such that the computer readable storage mediumhaving instructions stored therein includes an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks. Thecomputer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The various features and processes described above may be usedindependently of one another or may be combined in various ways. Allpossible combinations and sub-combinations are intended to fall withinthe scope of the invention. In addition, certain method or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto can be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically disclosed ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe disclosed example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the disclosed example embodiments.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which includes one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed partially, substantially, or entirelyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other example embodiments of the invention.All such modifications and variations are intended to be included hereinwithin the scope of the invention. While example embodiments of theinvention may be referred to herein, individually or collectively, bythe term “invention,” this is merely for convenience and does not limitthe scope of the invention to any single disclosure or concept if morethan one is, in fact, disclosed. The foregoing description detailscertain embodiments of the invention. It will be appreciated, however,that no matter how detailed the foregoing appears in text, the inventioncan be practiced in many ways. It should be noted that the use ofparticular terminology when describing certain features or aspects ofthe invention should not be taken to imply that the terminology is beingre-defined herein to be restricted to including any specificcharacteristics of the features or aspects of the invention with whichthat terminology is associated.

The embodiments illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other embodiments may be used and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of the invention. The Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

Although the invention(s) have been described in detail for the purposeof illustration based on what is currently considered to be the mostpractical and preferred implementations, it is to be understood thatsuch detail is solely for that purpose and that the invention is notlimited to the disclosed implementations, but, on the contrary, isintended to cover modifications and equivalent arrangements that arewithin the spirit and scope of the appended claims. For example, it isto be understood that the present invention contemplates that, to theextent possible, one or more features of any embodiment can be combinedwith one or more features of any other embodiment.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Moreover, plural instances may be provided forresources, operations, or structures described herein as a singleinstance. Additionally, boundaries between various resources,operations, program modules, engines, and/or datastores are somewhatarbitrary, and particular operations are illustrated in a context ofspecific illustrative configurations. Other allocations of functionalityare envisioned and may fall within a scope of various embodiments of theinvention. In general, structures and functionality presented asseparate resources in the example configurations may be implemented as acombined structure or resource. Similarly, structures and functionalitypresented as a single resource may be implemented as separate resources.These and other variations, modifications, additions, and improvementsfall within a scope of embodiments of the invention as represented bythe appended claims. The specification and drawings are, accordingly, tobe regarded in an illustrative rather than a restrictive sense.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment. Inaddition, it should be appreciated that any operation, element,component, data, or the like described herein as being based on anotheroperation, element, component, data, or the like can be additionallybased on one or more other operations, elements, components, data, orthe like. Accordingly, the phrase “based on,” or variants thereof,should be interpreted as “based at least in part on.”

What is claimed is:
 1. A computer-implemented method for fusing sensordata via metadata association, the method comprising: capturing firstsensor data using a first vehicle sensor and second sensor data using asecond vehicle sensor, the first vehicle sensor traversing at least aportion of a scan path that includes a 360 degree rotation; associatingfirst metadata with the first sensor data to obtain labeled first sensordata and second metadata with the second sensor data to obtain labeledsecond sensor data; calibrating a set of extrinsics for the firstvehicle sensor or the second vehicle sensor, the calibrated set ofextrinsics including rotational or translational transformation databetween the first vehicle sensor and the second vehicle sensor;performing, based at least in part on the calibrated set of extrinsics,a frame synchronization between the first sensor data and the secondsensor data to obtain a set of synchronized frames, wherein eachsynchronized frame includes a respective portion of the first sensordata and a corresponding respective portion of the second sensor data;executing, for each synchronized frame in the set of synchronizedframes, a metadata association algorithm on the labeled first sensordata and the labeled second sensor data; and generating, based at leastin part on an output of executing the metadata association algorithm,fused sensor data from the labeled first sensor data and the labeledsecond sensor data; and training a machine learning model or classifierbased on the fused sensor data.
 2. The computer-implemented method ofclaim 1, wherein the first sensor data is three-dimensional (3D) pointcloud data and the second sensor data is two-dimensional (2D) imagedata.
 3. The computer-implemented method of claim 2, wherein associatingthe first metadata with the first sensor data to obtain the labeledfirst sensor data comprises associating a 3D bounding box with a firstobject present in the 3D point cloud data, and wherein associating thesecond metadata with the second sensor data to obtain the labeled secondsensor data comprises associating a 2D bounding box with a second objectpresent in the 2D image data.
 4. The computer-implemented method ofclaim 3, wherein associating the first metadata with the first sensordata to obtain the labeled first sensor data and the second metadatawith the second sensor data to obtain the labeled second sensor datafurther comprises associating a first classification with the firstobject and a second classification with the second object.
 5. Thecomputer-implemented method of claim 2, wherein each synchronized frameis associated with i) a respective set of 3D labels representative of arespective set of objects present in the respective portion of the 3Dpoint cloud data and ii) a respective set of 2D labels representative ofa respective set of objects present in the respective portion of the 2Dimage data.
 6. The computer-implemented method of claim 5, whereinexecuting, for each synchronized frame in the set of synchronizedframes, the metadata association algorithm comprises determining, foreach synchronized frame, a respective set of similarity scores, whereineach similarity score in the respective set of similarity scorescorresponds to a respective 3D label and a respective 2D label andrepresents a quantitative value indicative of a likelihood that therespective 3D label and the respective 2D label represent a same objectin the synchronized frame.
 7. The computer-implemented method of claim6, wherein executing, for each synchronized frame in the set ofsynchronized frames, the metadata association algorithm furthercomprises: generating, for each synchronized frame, a respective scoringmatrix comprising the respective set of similarity scores; anddetermining that a particular 3D label and a particular 2D label form amatching label pair representative of a same particular object in thesynchronized frame, wherein determining that the particular 3D label andthe particular 2D label form the matching label pair comprisesdetermining that a particular similarity score between the particular 3Dlabel and the particular 2D label is greater than each other similarityscore associated with the particular 3D label and each other similarityscore associated with the particular 2D label.
 8. Thecomputer-implemented method of claim 7, wherein the fused sensor datacomprises an indication of the matching label pair.
 9. Thecomputer-implemented method of claim 1, further comprising: receivingfirst timing data from the first vehicle sensor and second timing datafrom the second vehicle sensor, wherein the first timing data isindicative of a first amount of time required by the first vehiclesensor to capture the first sensor data and the second timing data isindicative of a second amount of time required by the second vehiclesensor to capture the second sensor data.
 10. The computer-implementedmethod of claim 9, wherein performing the frame synchronizationcomprises performing the frame synchronization based at least in part onthe calibrated set of extrinsics, the first timing data, and the secondtiming data.
 11. A system for fusing sensor data via metadataassociation, the system comprising: at least one processor; and at leastone memory storing computer-executable instructions, wherein the atleast one processor is configured to access the at least one memory andexecute the computer-executable instructions to: capture first sensordata using a first vehicle sensor and second sensor data using a secondvehicle sensor, the first vehicle sensor traversing at least a portionof a scan path that includes a 360 degree rotation; associate firstmetadata with the first sensor data to obtain labeled first sensor dataand second metadata with the second sensor data to obtain labeled secondsensor data; calibrate a set of extrinsics for the first vehicle sensoror the second vehicle sensor, the calibrated set of extrinsics includingrotational or translational transformation data between the firstvehicle sensor and the second vehicle sensor; perform, based at least inpart on the calibrated set of extrinsics, a frame synchronizationbetween the first sensor data and the second sensor data to obtain a setof synchronized frames, wherein each synchronized frame includes arespective portion of the first sensor data and a correspondingrespective portion of the second sensor data; execute, for eachsynchronized frame in the set of synchronized frames, a metadataassociation algorithm on the labeled first sensor data and the labeledsecond sensor data; and generate, based at least in part on an output ofexecuting the metadata association algorithm, fused sensor data from thelabeled first sensor data and the labeled second sensor data; and traina machine learning model or classifier based on the fused sensor data.12. The system of claim 11, wherein the first sensor data isthree-dimensional (3D) point cloud data and the second sensor data istwo-dimensional (2D) image data.
 13. The system of claim 12, wherein theat least one processor is configured to associate the first metadatawith the first sensor data to obtain the labeled first sensor data byexecuting the computer-executable instructions to associate a 3Dbounding box with a first object present in the 3D point cloud data, andwherein the at least one processor is configured to associate the secondmetadata with the second sensor data to obtain the labeled second sensordata by executing the computer-executable instructions to associate a 2Dbounding box with a second object present in the 2D image data.
 14. Thesystem of claim 13, wherein the at least one processor is configured toassociate the first metadata with the first sensor data to obtain thelabeled first sensor data and the second metadata with the second sensordata to obtain the labeled second sensor data by executing thecomputer-executable instructions to associate a first classificationwith the first object and a second classification with the secondobject.
 15. The system of claim 12, wherein each synchronized frame isassociated with i) a respective set of 3D labels representative of arespective set of objects present in the respective portion of the 3Dpoint cloud data and ii) a respective set of 2D labels representative ofa respective set of objects present in the respective portion of the 2Dimage data.
 16. The system of claim 15, wherein the at least oneprocessor is configured to execute, for each synchronized frame in theset of synchronized frames, the metadata association algorithm byexecuting the computer-executable instructions to determine, for eachsynchronized frame, a respective set of similarity scores, wherein eachsimilarity score in the respective set of similarity scores correspondsto a respective 3D label and a respective 2D label and represents aquantitative value indicative of a likelihood that the respective 3Dlabel and the respective 2D label represent a same object in thesynchronized frame.
 17. The system of claim 16, wherein the at least oneprocessor is further configured to execute, for each synchronized framein the set of synchronized frames, the metadata association algorithm byexecuting the computer-executable instructions to: generate, for eachsynchronized frame, a respective scoring matrix comprising therespective set of similarity scores; and determine that a particular 3Dlabel and a particular 2D label form a matching label pairrepresentative of a same particular object in the synchronized frame,wherein determining that the particular 3D label and the particular 2Dlabel form the matching label pair comprises determining that aparticular similarity score between the particular 3D label and theparticular 2D label is greater than each other similarity scoreassociated with the particular 3D label and each other similarity scoreassociated with the particular 2D label.
 18. The system of claim 17,wherein the fused sensor data comprises an indication of the matchinglabel pair.
 19. The system of claim 11, wherein the at least oneprocessor is further configured to execute the computer-executableinstructions to: receive first timing data from the first vehicle sensorand second timing data from the second vehicle sensor, wherein the firsttiming data is indicative of a first amount of time required by thefirst vehicle sensor to capture the first sensor data and the secondtiming data is indicative of a second amount of time required by thesecond vehicle sensor to capture the second sensor data, wherein the atleast one processor is configured to perform the frame synchronizationby executing the computer-executable instructions to perform the framesynchronization based at least in part on the calibrated set ofextrinsics, the first timing data, and the second timing data.